Automatic correction of keystone distortion and other unwanted artifacts in projected images

ABSTRACT

A system for manual or automatic correction of geometric or video image distortions introduced by projection onto obliquely angled or imperfect surfaces. Sensors may be disposed at the projector itself, at the projected image surface, or in a portable remote-control unit. Corrections may be applied as part of an original set-up process, or dynamically as the configuration changes. Image-stabilization techniques are utilized where applicable, as is the use of test patterns and the incorporation of “helper” signals into the video image. Tactile-sensing capabilities provide an interactive environment, and various packaging configurations for projection units and sensor provisions are disclosed.

REFERENCE TO RELATED APPLICATION

This application claims priority from U.S. Provisional PatentApplication Ser. No. 61/439,208, filed Feb. 3, 2011, the entire contentof which is incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates to image and video projection systems, and, moreparticularly, to a system for correcting geometric and distortionsintroduced by projection onto imperfect surfaces.

BACKGROUND OF THE INVENTION

With the advent of super bright and ultra-bright light-emitting diodesand portable electronic devices that are too compact to provide displayswhich show a great deal of information, projected video and still imageswill become more popular. At the same time, since it might not be knownupon which surface the light will be projected, certain undesirableimage artifacts and distortions, such as keystone effects, will need tobe addressed. Although there are many proposed solutions, includingpatents which are directed to keystone correction, they all makeassumptions about the relationship between the projector and theprojected image; if these assumptions are incorrect, a geometricallyunacceptable projection may still result. This invention corrects forkeystone effects and other unwanted image artifacts and deficienciessuch as brightness, contrast, color uniformity, etc., in a projectedimage, regardless of the surface chosen for projection.

SUMMARY OF THE INVENTION

The instant invention comprises a video image projection system capableof sensing and correcting for image distortions, such as keystoneeffects and image artifacts and deficiencies, which are introduced byimperfect projection configurations. In particular, provisions areincluded for off-angle projection, or image projection onto imperfectsurfaces.

Projected image correction apparatus constructed in accordance with theinvention includes a device with a display generator and a projectorprojecting an image onto a surface. A sensor views the projected imageand communicate information to the device enabling the display generatorto correct for keystone effects or incorrect aspect ratio; variations inbrightness, contrast, color rendition, saturation, hue, focus,sharpness, or movement of the projector. The device may be a videoprojector, camcorder, portable computer, smart phone.

The sensor, which may be a two-dimensional image sensor, may be disposedin the projector device, or in a hand-held remote control unit whichcommunicates the information to the device through a wirelessconnection. The correction may occur in conjunction with an initialprojection, in response to a user input, or on a continuous basis.

The projection device may include a detachable display screen providingthe surface, for example a detachable rigid or semi-rigid panel or aroll-out screen. The screen may include corner or edge indicia to assistthe sensor in determining the periphery of the screen. Hardware orsoftware may be included to generate a test pattern for correctionpurposes.

With the image geometry sensor disposed on or in the projection device,the unit senses geometric and video distortions, and provides correctiveinformation to projection circuitry, enabling the unit to manipulate theshape of the projected image so as to produce a “squared” image withcorrections to video parameters such as brightness, color rendition, andso forth. The corrections may be either manually applied, orautomatically applied by the projection circuitry.

Alternatively the sensor may be disposed on or in a portable remotecontrol unit in wireless communication with the projection unit. Thissimplifies the adjustments to be applied to the projected image wherethere is a desire to apply these corrections from the point of view ofthe audience, or when the projector itself may not be in a fixedlocation. If a detachable projection screen is utilized it may includesensing provisions located either on the projection unit, oralternatively on the projection screen itself.

One preferred implementation of the invention includes a display bufferstoring a raw or ‘desired’ image to be seen by a viewer. A projectionbuffer is used to store a version of the raw image to be projected ontothe surface, and a sensor buffer is used to store an image of theprojected image as viewed by the sensor. The display processor isoperative to compare the image stored in the sensor buffer to the imagestored in the display buffer and adjust the image stored in theprojection buffer so that the projected image seen by the sensorcorresponds to the image stored in the display buffer. The displayprocessor may be operative to perform one or more of the followingadjustments to the image stored in the projection buffer: flipping,stretching, luminance or color curve adjustment, color replacement,pixel interpolation, or rotation.

Further aspects of the invention enables the projected image to functionas a “touch screen,” enabling a user to “select” points on the displayedimage, enlarge and reduce dimensions, scroll, turn pages, and so forth.This feedback feature can be used to “stretch” the corners of the image(much like the “rubber-band” feature in graphic and drawing softwareenables the manipulation of the shape of an object), thereby enablingthe user to “click-and-drag” the corners of the projected image tomanually correct for keystone effects, or to assist in theauto-correction process for geometric correction. These embodimentsinclude gesture recognition apparatus operative to recognize a gesturemade by a finger, hand or other body part overlying the projected imagein the view of the sensor, and control the device or projected image inresponse to the recognized gesture.

Assuming the projected image has x and y coordinates associatedtherewith, the gesture recognition apparatus is operative to determinethe coordinates of a finger, hand or other body part overlying theprojected image in the view of the sensor. The system is then operativeto compare the position or movement thereof a finger, hand or other bodypart overlying the projected image to corresponding coordinatesassociated with the projected image to control the device or theprojected image. The sensor may include an infrared imaging capabilityto assist in detecting a finger, hand or other body part overlying theprojected image.

A tactile sensor may be provided to generate a wireless signal if apoint of the surface is touched by a user. A wireless receiver receivesthe wireless signal, with the system being operative to receive theimage viewed by the sensor, determine the coordinates of the pointtouched by the user in response to receipt of the wireless signal, andcompare the coordinates to the coordinates of the projected image toselect a region of the projected image associated with the point. Thewireless signal may be an optical signal such as an infrared signal,with the sensor is operative to detect the optical signal, or an RF oracoustical signal.

The various embodiments of the invention provide for numerous features,including automatic or manual correction of geometric distortions, videoparameters, and screen texture and color. Provisions also are includedfor automatic “horizontal level-sensing,” image-stabilization, and atactile-sensing interactive environment, as well as the use ofinternally-generated test patterns and an artificial image-frame toassist in manual and automatic correction of image deficiencies. Thedisclosure anticipates a variety of packaging configurations, fromsmall, portable projection devices (such as cellular phones or PDAs), tolarge scale stand-alone video projectors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts a first embodiment of the invention, in which the sensordevice is disposed at the projector unit;

FIG. 1B depicts an alternative embodiment of the invention, in which thesensor device is disposed on portable remote-control unit incommunication with the projector unit;

FIG. 2 depicts an alternative embodiment of the invention, in which thesensor device is disposed on the projector unit, and a detachable screenis used for the image projection surface;

FIG. 3 depicts a functional block diagram displaying many of thefeatures of the preferred embodiment of the invention;

FIGS. 4A and 4B are simplified drawings which illustrate the way inwhich the correction process operates;

FIGS. 5A and 5B are simplified drawings which illustrate the way inwhich the invention provide image stabilization;

FIG. 6 shows a device such as a smart phone having a projector producingan image on a surface and a sensor viewing the projected image alongwith gesture recognition which determines the position and/or movementof a finger, hand, or other body part for control purposes; and

FIG. 7 illustrates how selection of a part of a projected image may beimplemented with a fingertip device that generates a wireless signalwhen a surface is touched.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1A illustrates one embodiment of the invention including an imageprojector 104 and sensor 108 integrated into a single unit 102. For thepurposes of this invention, “image projector” should be taken to meanstill or moving pictures (i.e., video), regardless of aspect ratio orresolution. The invention is not limited in terms of projectortechnology, which includes light-transmissive (i.e., LCD) andlight-reflective (i.e., DLP) approaches. “Sensor” should be taken tomean any type of suitable image-gathering device, preferably atwo-dimensional image sensor based upon charge-coupled devices (CCDs),for example. Although a relatively high resolution color sensor ispreferred, that is not necessary to the basic implementation of theinvention. To correct artifacts such as keystone effects alone, forexample, a relatively low-resolution image sensor may be used, even amonochrome sensor. Also the “unit” 102 should be taken to include adedicated or stand-alone projector adapted for connection to a source ofimagery to be projected, or any device that would benefit from aprojected display, including laptop and tablet computers, smart phones,camcorders, and so forth.

Continuing the reference to FIG. 1A, projector 104 produces an image 106on a surface to be viewed by an audience. The image 106 may includevarious undesirable artifacts, including keystone effects or otheraspect-ratio issues; brightness, contrast, color, focus or sharpnessvariations, and so forth. To alleviate such problems, this inventionutilizes an image sensor which observes the projected image to obtain atrue, real-time picture including any undesirable artifacts that may bepresent. Information about the projected image is then communicated to agraphics processing unit (GPU) or other electronic circuitry within theprojection unit to correct for the artifacts detected, as discussed infurther detail below. In all embodiments, the correction may occur at agiven time during start-up or by pressing a “correct” button, forexample, or adjustments may automatically occur at present intervals orin real time to account for changes in position or lighting over time.

In FIG. 1A, the projector and sensor are integrated, such that theprojected image is corrected from the perspective of the projectoritself. There may be situations, however, where it is desirable tocorrect the image from a perspective other than that of the projector,such as that seen by a particular viewer. FIG. 1B illustrates analternative embodiment of the invention wherein the image sensor/camerais housed in a hand-held remote control unit 110 which communicateswirelessly to projection device 102. The remote 110 may simply be animage correction device, including a still or video camera 112; however,in the preferred configuration, the remote device 110 fauns the sameremote control device used to change images and control the projector102. With this embodiment the remote unit may be operated at a desiredlocation associated with an audience or selected viewer, therebycorrecting for artifacts relative to a particular viewing location. Inaddition, this embodiment allows for corrections to be implemented whenthe projector 102 may not maintain a fixed position.

Thus, in each of the embodiments disclosed herein, corrections can beapplied to provide the desired image as viewed from various points ofview: rectilinear or “squared” image as seen by the projection source;“squared” image as seen by the audience; or “squared” image as seen by adynamically changing point of view. The applicability of the inventionis not limited by the size of the projected image, the texture or natureof the surface onto which the image is to be projected, or theparticular position of the projection unit relative to the surface ontowhich the image is to be projected, and any variations in thesevariables should be considered to be within the scope of the invention.

FIG. 2 illustrates of the applicability of the invention to a portableelectronic device 202 having a detachable screen 204 and a built-inprojector projecting an image 206 onto the screen 204. The image sensorcan be built into the device 202, or in a remote unit as discussed withrespect to FIGS. 1A and 1B. Feet 210 may be used to stabilize the screen204 at a desired distance from the projecting device 202. The projectingdevice 202 may be any kind of portable electronic device, including alaptop or tablet computer; MP3/video player, a device utilizing adisplay for video content or controls, or the like. The screen 204 mayhave corner dots 208 or other indicia, such that regardless of where itis placed relative to the projecting unit 202, the sensor may betterdiscern the outer periphery of the screen 204 and project the image tocompletely fill that screen, if desired. Alternatively, the sensors 208may be connected to the projector unit through a communication link,thereby providing direct feedback from the surface as adjustments aremade.

As opposed to a rigid screen which detaches from the portable electronicdevice as shown in the upper portion of FIG. 2, a flexibleunrolling-type screen may be used, much like the portable screens usedwith slide projectors and movie cameras of the past. This would allowthe screen to be installed and removed from the portable electronicdevice as a small cylindrical tube. Alternatively, of course, theprojection device 202 may project onto a wall or other surface.

FIG. 3 illustrates a simplified block diagram of the preferredembodiments. In essence, a ‘feedback loop’ 300 is established inaccordance with the invention, whereby the projected image 302 isdetected and analyzed by the sensor/camera 304 and corrected, eitherautomatically and/or in accordance with user viewing preferences. Adisplay 314 is shown because the invention does not preclude the use ofa display panel other than or in addition to the projected image. Acomputer or smart phone, for example, would typically have both adisplay screen and a projector. To enhance operation, projecting devicemay be equipped with the capability of producing an image at 106 whichis specifically intended for image correction; that is, it may have atest pattern, with edges, color or brightness better “seen” by thesensor. These kinds of test materials are known to those skilled in theart of video camera set-up.

There may be some difficulties involved in the detection of the cornersof the projected images by the sensor. For example, when the sensor isco-located within the projector, if a perpendicular to the surface thatis to receive the image is at an oblique angle relative to theprojection axis, then the amount of light reflected back towards thesensors will be reduced significantly. Due to such circumstances, it isadvantageous for the sensor to have sufficient sensitivity to detect lowlevels of reflected light. In addition, the amount of light reflectedwill be affected by many other factors depending on the smoothness ofthe surface, the color of the surface, the brightness of the projectedimage, etc. In such cases, the use of “fuzzy logic” as applied incommercially available consumer video cameras may be utilized to provideguidance to the correction process, as described herein below.

There are several options which may be implemented to improve theutility and performance of the system. For example, provisions may beincluded to automatically identify “level” or true horizontal, to assistin the “squaring” process. Additionally, appending a “frame” linesurrounding the projected image pixels can assist in detecting when theprojected image is “square,” and also assist in the “leveling” process.For example, in the embodiment depicted in FIG. 2, the corner detectioncan be utilized to locate the “edge” of the image, either from the“frame” markings, or from detection of the apparent image edge based oncontent (or lack thereof), and the information then can be used toautomatically adjust the geometry of the projected image.

In addition, internally generated “test patterns”, such as thetraditional “convergence pattern” “color bars,” etc. and other testsignals utilized in broadcast and other professional applications, canassist the alignment of the system, in much the same way that thesesignals are utilized to align and calibrate the geometric, luminance,and chrominance characteristics of professional cameras and videomonitors. Many of the same techniques applied in systems utilized forautomatic alignment and matching of cameras and monitors couldadvantageously be applied for aligning the projected image system of theinstant invention.

With the use of a standardized test signal, such as the “convergencepattern” signal, it would be possible to detect the texture and/orimperfections of the surface on which the image is to be projected.Knowing this information, corrections can be applied at whatever levelof detail is desired by the user, and even complicated distortions, suchas would be encountered in projecting onto a brick wall or hangingdrapes, could be applied to optimize the visible image. Similarly,corrections to luminance and/or chrominance imperfections can beapplied, to optimize the visible image at whatever level of detail isdesired by the user. In the case of uneven surfaces, particularly thosewith repeating patterns such as brick walls, etc., a white oruniform-color display may be projected, enabling the sensor to gather animage of the surface texture which may then be used by the graphicsprocessor to “subtract off” irregularities in luminance, color, focus,sharpness, and so forth.

FIGS. 4A and 4B are simplified drawings which better illustrate the wayin which the correction process operates. In the preferred embodimentsdifferent buffer memories are used for different purposes, namely for“raw” and “corrected” and images. In reference to FIG. 4A, the image inthe display buffer represents a raw image to be displayed without anyundesirable artifacts generated through color or luminance variations orprojection onto uneven or imperfect surfaces. If the raw image is simplytransferred to the projection buffer used to project the image, theactual image seen by the sensor as shown in the sensor buffer suffersfrom horizontal and vertical keystone distortion and side-to-sidewashout (depicted with gray tone variation) due to uncontrollablelighting effects and/or projected surface unevenness.

To correct for these distortions, a graphics processor analyzes theimage in the sensor buffer and compares is to the desired raw image inthe display buffer. The graphics processor modifies the image in theprojection buffer so that the image in the sensor buffer matches the rawimage in the display buffer, at least as practical under thecircumstances. Typically the projection buffer will generate anintentionally distorted and/or mirror image of the raw image, such thatwhen the intentionally distorted or mirror image is projected, the imagein the sensor buffer more closely resembles the desired image in thedisplay buffer. The graphics processor may use various techniques tocreate the intentionally distorted and/or mirror image of the raw image,including flipping, stretching, luminance or color curve adjustment,color replacement, pixel interpolation, rotation and so forth.

To correct for movement of the projector the invention may incorporateimage-stabilization technology. In many consumer cameras and camcorders,provisions are included for automatic image-stabilization, to mitigatethe artifacts introduced by the user while simply trying to hold thecamera steady. Mechanical provisions, such as gyroscopes, may beincluded, or the correction may be based on electronic methods. In thecase of the electronic approach to this problem, movement is detected bycomparison of consecutive frames (or fields), to determine which objectsare moving, and which movements actually are related to unsteadiness bythe holder. Movement of the entire frame of the image is attributed tounsteadiness, while movement of individual objects relative to otherobjects is attributed to true motion by these objects. Movement of theentire image frame can be canceled out by correcting for the detectedmotion, leading to a more stable image. Importantly, this also providesan opportunity to apply noise reduction, to improve the quality of theimage as projected.

FIGS. 5A and 5B illustrate this aspect of the invention. In FIG. 5A, theentire image in the sensor buffer is shifted towards the upper left,indicating that the projector has moved. This may occur with anyhand-held device such as a smart prone projector being used in a movingcar, as one example of many. The graphics processor detects thismovement through comparison of the image in the display buffer to theshifted image in the sensor buffer. To stabilize the image, the image inthe projection buffer is intentionally shifted to compensate for themovement such that the images in the sensor and display buffers nowcorrespond. In the event the graphics processor is unable to compensatefor the motion if its is extreme or otherwise unexpected, the projectionbuffer may be selectively turned off or the frame rate reduced tofacilitate movement compensation.

A further aspect of the invention enables the projected image tofunction as a “touch screen,” enabling a user to “select” points on thedisplayed image, enlarge and reduce dimensions, scroll, turn pages, andso forth, much like operations now possible with the iPhone and pointingdevices such as a mouse or touchpad. This feedback feature can be usedto “stretch” the corners of the image (much like the “rubber-band”feature in graphic and drawing software enables the manipulation of theshape of an object), thereby enabling the user to “click-and-drag” thecorners of the projected image to manually correct for keystone effects,or to assist in the auto-correction process for geometric correction.

Since the invention incorporates a sensor operative to look at theprojected image, the sensor may be used to detect a finger, hand orother body part in the projection and interpret position or movementsmade as selections of control inputs. FIG. 6 shows a device 602 such asa smart phone having a projector 604 producing an image 605 on a surfaceand a sensor 606 viewing the projected image. The hand 610 of a userintersects the projected image, and the user is using their fingers toenlarge a region 612 of the projected image. This is possible becausethe x-y pixel coordinates of the projected image are known by thegraphics processor. Even if the graphics processor has manipulated orcorrected the projected image, the x-y coordinates are known throughcomparison(s) with the raw image in the display buffer.

Continuing the reference to FIG. 6, the coordinates of the hand 610 (orother body part) may also be determined in several ways. Since theprojected image should “look like” the raw image in the display buffer,gross differences in area seen by the sensor may be presumed to be anobject in the path of the projected light, particularly if the object(i.e., hand or finger) is moving. In addition, if the sensor is able todetect in the infrared (which is often the case), the sensor (orseparate sensor) can be used to identify warm objects that may bepresumed to be body parts. One a body part is detected, position andmovements may be interpreted with gesture recognition hardware orsoftware known to those of skill in the art, and my comparing thecoordinates of the gesture with the underlying projected display,control inputs such as enlarging, reducing, scrolling, page turning,opening, closing, and so forth may be implemented, much like a touchscreen.

Selection presents a technical challenge, since the user is touching theprojected image on a remote surface as opposed to a display screen.However, this problem may be solved if the user 710 wears a fingertipdevice 712 shown in FIG. 7. This device includes button at the tip ofthe finger which is activated when a surface is touched. Activation ofthe button causes the transmission of a wireless signal 716 received bythe projector device 702. Using the location/movement detectioncapabilities discussed with reference to FIG. 6, the coordinates of thetouched point may be determined and compared to the x-y coordinates ofthe projected images to facilitate a selection of the point, area,region or icon underlying the touched point. The wireless signalgenerated by the fingertip device may be an RF or acoustical signal or,alternatively, an infrared light may be generated which is picked up bythe sensor 706.

The selection device need not be worn by a person and may alternativelybe put on the end of a pointed used to tap the surface. As a furtheralternative, pressure-sensitive surfaces for the projected images, ortouch-screen frames can provide information as to where a user may betouching the image. Various applications can benefit from such tactileor other feedback techniques enabling the various embodiments tofunction in an interactive environment. These kinds of signals can beused to trigger other events, such as advancing the slides in apresentation, or activating a link to another portion of a controlprogram or a web site.

1. Projected image correction apparatus, comprising: a device includinga display processor and a projector operative to project an image onto asurface; a sensor operative to view the projected image on the surface;and electronic circuitry operative to communicate information to thedisplay processor about the image viewed by the sensor, enabling thedisplay processor to correct for one or more of the following defects inthe projected image: keystone effects or incorrect aspect-ratio;brightness or contrast variations; incorrect color rendition, saturationor hue; poor focus or sharpness; or motion artifacts caused by movementof the device.
 2. The apparatus of claim 1, wherein: the device is avideo projector; and the projector is operative to project a video imageonto the surface.
 3. The apparatus of claim 1, wherein the device is aportable computer.
 4. The apparatus of claim 1, wherein the device is asmart phone.
 5. The apparatus of claim 1, wherein the device is acamcorder.
 6. The apparatus of claim 1, wherein the sensor is atwo-dimensional image sensor.
 7. The apparatus of claim 1, wherein thesensor is disposed in the device.
 8. The apparatus of claim 1, wherein:the sensor is disposed in a hand-held remote unit operative to controlcertain functions of the device; and the hand-held remote unitcommunicates the information to the device through a wirelessconnection.
 9. The apparatus of claim 1, wherein the correction occursin conjunction with an initial projection.
 10. The apparatus of claim 1,wherein the correction occurs in response to a user input.
 11. Theapparatus of claim 1, wherein the correction automatically occurs on anoccasional or continuous basis.
 12. The apparatus of claim 1, whereinthe display generator is further operative to generate a test pattern tobe viewed by the sensor for correction purposes.
 13. The apparatus ofclaim 1, wherein the device includes a detachable or roll-out displayscreen providing the surface.
 14. The apparatus of claim 1, furtherincluding: a display screen providing the surface; and wherein: thedisplay screen includes corner or edge indicia to assist the sensor indetermining the periphery thereof.
 15. The apparatus of claim 1, furtherincluding: a display buffer storing a raw image to be seen by a viewer;a projection buffer storing a version of the raw image to be projectedonto the surface; a sensor buffer storing an image of the projectedimage as viewed by the sensor; and wherein: the display processor isoperative to compare the image stored in the sensor buffer to the imagestored in the display buffer and adjust the image stored in theprojection buffer so that the projected image seen by the sensorcorresponds to the image stored in the display buffer.
 16. The apparatusof claim 15, wherein the display processor is operative to perform oneor more of the following adjustments to the image stored in theprojection buffer: flipping, stretching, luminance or color curveadjustment, color replacement, pixel interpolation, and rotation. 17.The apparatus of claim 1, further including: gesture recognitionapparatus operative to recognize a gesture made by a finger, hand orother body part overlying the projected image in the view of the sensor;and control the device or projected image in response to the recognizedgesture.
 18. The apparatus of claim 1, wherein: the projected image hasx and y coordinates associated therewith; and gesture recognitionapparatus operative to determine the coordinates of a finger, hand orother body part overlying the projected image in the view of the sensor;and apparatus operative to compare the position or movement thereof afinger, hand or other body part overlying the projected image tocorresponding coordinates associated with the projected image to controlthe device or the projected image.
 19. The apparatus of claim 1,wherein: the projected image has x and y coordinates associatedtherewith; the sensor includes an infrared imaging capability operativeto detect a finger, hand or other body part overlying the projectedimage; and gesture recognition apparatus operative to determine thecoordinates of the finger, hand or other body part detected by thesensor; and apparatus operative to compare the position or movementthereof a finger, hand or other body part overlying the projected imageto corresponding coordinates associated with the projected image tocontrol the device or the projected image.
 20. The apparatus of claim 1,wherein: the projected image has x and y coordinates associatedtherewith; further including a tactile sensor operative to generate awireless signal if a point of the surface is touched by a user; awireless receiver operative to receive the wireless signal; andapparatus operative to receive the image viewed by the sensor, determinethe coordinates of the point touched by the user in response to receiptof the wireless signal, and compare the coordinates to the coordinatesof the projected image to select a region of the projected imageassociated with the point.
 21. The apparatus of claim 20, wherein: thewireless signal is an optical signal; and the sensor is operative todetect the optical signal.
 22. The apparatus of claim 20, wherein: thewireless signal is an infrared signal; and the sensor is operative todetect the infrared signal.
 23. The apparatus of claim 20, wherein: thewireless signal is an RF signal.